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Universal remote control unit with automatic appliance identification and programming 



FIELD OF THE INVENTION 

The invention relates to remote control units that control appliances, home 
electronic equipment and the like. 

5 BACKGROUND OF THE INVENTION 

Remote control units are commonly available today to control many devices, 
especially consumer electronic devices. Most of the public is familiar with a remote control 
unit (often simply referred to as a "remote") for controlling their television sets and VCRs. 
Other devices such as DVD players, CD players and other home stereo equipment are 

10 controllable via remote control devices. Other devices (such as appliances) and systems (such 
as a home alarm system) may also be controlled by remote control devices. 

Remote control units typically operate by emitting a sequence of infrared 
pulses that are received by the controlled device. The particular sequence provided reflects an 
encoded command (such as on, off, adjust volume, change channel) that is recognized by the 

15 controlled device. (An "encoded command" transmitted between a remote and a particular 
device will alternatively be referred to as a "command protocol".) When received by the 
controlled device, the command protocol is recognized and executed by the device. However, 
when the command protocol for a particular device is received by a different device, 
generally nothing will occur because the command protocol does not represent a command 

20 for that device when decoded. 

There are also "universal" remote control units that can be programmed to 
control a number of different devices, such as a television and a VCR. For such a universal 
remote, there are a number of buttons that enable the user to select the device to be 
controlled. (Alternatively, one selection button may be used to scroll through the controlled 

25 devices.) After the user selects a device, any commands selected are emitted as a command 
protocol that is recognized by the selected device. For example, if the user selects "VCR" on 
a programmed universal remote, the encoded command (command protocol) emitted by the 
remote will be recognized by the VCR. If the user then selects the command "channel 7" via 
the remote, the channel signal sent by the remote will be encoded to change the channel 
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setting on the VCR. Decoding of the encoded channel command by the television will 
generally not be recognized and the television will remain tuned to the same channel. 

As noted, universal remotes may be programmed to control a number of 
devices. Typically a universal remote is sold loaded with a comprehensive database of the 
command protocols for many available devices. The universal remote may then be 
programmed via a programming sequence so that the particular command protocols for 
devices found in the user's home may be selected by the user. For example, the universal 
remote may be programmed manually by the user by inputting a particular reference number 
(sometimes referred to as a device's "appliance code") that corresponds to a device in the 
user's home. The universal remote includes in the database the appliance codes and the 
associated command protocols for those devices that may be controlled by programming the 
remote. By inputting the appliance code for a particular device, the remote searches the 
database for the appliance code and the associated command protocols and uses the 
command protocols when control commands are subsequently input by the user via the 
remote. The programming sequence may be conducted for a number of different devices 
utilizing the corresponding appliance code for each such device. The programming sequence 
may then also include assigning the command protocols for each programmed device to one 
of the selection buttons (or the spots assignable using a single button). 

Alternatively, some universal remote control units provide a procedure to 
determine the appliance code of a device in the user's home by sequentially emitting one or 
more encoded commands for each of the devices in the remote's database and having the user 
stop the search when the user sees that a command that properly controls the user's device is 
emitted. 

Among the disadvantages of the existing art regarding universal remotes, the 
user must manually select the device that is to be controlled. Thus, the user must first select 
the device by depressing a button on the remote, and then select the control command to be 
sent to the device. This requires a number of manipulations that are inconvenient to the user. 
For example, a user may be recording a program on channel 2 on the VCR and watching a 
program on the channel 4 on the television. In such an instance, the user may typically have 
the television selected on the universal remote, so that the volume may be controlled. If the 
user wants to change the station being recorded on the VCR, the user must select "VCR" on 
the remote, depress keys on the remote to change the channel on the VCR, and then select 
4 TV" on the remote so that the volume on the TV may once again be controlled. 
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In addition, as described above, currently available universal remotes need 
some type of user involvement during programming. Either the user must input the appliance 
codes of the devices during the programming sequence or, if the remote automatically emits 
control commands for all of the appliances in its database, the user must indicate when a 
control command is emitted that controls one of the user's devices. Both of these procedures 
add to the complexity of the universal remote, which can often render it unusable for the 
average consumer. 



10 SUMMARY OF THE INVENTION 

It is an objective of the current invention to provide a remote control unit and 
system that automatically determines which device the user intends to control and 
automatically switches so that the command protocol corresponding to that selected device 
are emitted by the remote when the user provides a control command to the remote. It is also 

1 5 an objective to provide a remote control unit and system that automatically determines the 
appliance code (or otherwise determines the appropriate control protocol) for the selected 
device, so that no programming by the user is required. In particular, it is an objective of the 
invention to provide such automatic programming of devices and automatic switching 
between devices when a number of such devices are located in one room, where the 

20 programming of numerous devices and switching between numerous devices can be 
extremely tedious and cumbersome. 

In accordance with these objectives, the invention comprises a remote that is 
capable of controlling a number of devices. The remote acquires identification data from a 
particular device at which it is pointed. The data is processed to determine command 

25 protocols associated with the particular device. Control commands for the particular device 
input to the remote by a user are formatted according to the command protocols associated 
with the particular device. 

The invention further comprises a method for controlling one particular device 
selected from among a number of devices based upon a selection direction. Identification 

30 data is acquired from the particular device. Command protocols associated with the particular 
device are determined using the identification data. Control commands for the particular 
device are formatted according to the determined command protocols for the particular 
device. 
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The invention further includes a system comprised of a number of devices that 
are controllable via a remote. The remote acquires identification data from the particular 
device among the number of devices at which the remote is pointed. Each device of the 
number of devices may, for example, emit identification data identifying the respective 
5 device. In that case, identification data of the particular device at which the remote is 

pointing is selected by the remote. Alternatively, the remote itself may initiate acquisition of 
the identification data of the particular device at which it is pointed. The identification data 
for the particular device is processed by the remote to determine command protocols 
associated with the particular device. Control commands for the particular device input to the 
10 remote by a user are formatted according to the command protocols associated with the 
particular device. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a representative view of an embodiment of the remote control unit 
1 5 and system of the present invention; 

Fig. 1 a is a more detailed view of the remote control unit of Fig. 1 ; 

Fig. lb is a representation of a number of components comprising the remote 

of Fig. la; 

Fig. 2 is a detailed view of a remote control unit according to a second 
20 embodiment of the invention; 

Fig. 2a is a representation of a number of components comprising the remote 

of Fig. 2; 

Fig. 3 is a view of an image captured by the remote of Fig. 2; 
Fig. 4 is a representative view of a portion of a database of templates stored in 
25 the remote of Fig. 2; and 

Fig. 5 is a flowchart of an embodiment of a method in accordance with the 
present invention. 

DETAILED DESCRIPTION 
30 Referring to Fig. 1, a universal remote control unit 10 (simply referred to as a 

"remote" hereinafter) is shown that incorporates an embodiment of the invention. Remote 10 
controls a number of devices, shown in Fig. 1 as television 12, VCR 14, stereo 
tuner/amplifier 16 and CD player 18. Television 12 and VCR 14 are electrically connected so 
that recorded programs may be played on the VCR 14 and viewed via the television 12 when 
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appropriate settings and commands are input to the devices, as is well-known in the art. 
Likewise, stereo timer/amplifier 16 and CD player 18 are electrically connected, so that CDs 
may be inserted in the CD player 18 and amplified via the stereo tuner/amplifier 18 for 
listening. 

5 As will be described further below, after remote 10 is utilized by the user to 

provide a control command to one of the devices in the manner described below according to 
the invention (which involves automatic selection and programming of the user's intended 
device), the remote emits signals according to the command protocol for the selected device. 
Fig. la shows a more detailed exemplary embodiment of remote 10, having various buttons 

10 providing a user interface where the user can input commands for a selected device. Among 
other things, remote 10 includes a power button 10a, volume control buttons 10b, channel 
input buttons 10c, channel up/down buttons lOd and VCR control buttons lOe (some of 
which may also be used to control CD player 18). When television 12 is the selected device, 
for example, depressing volume control buttons 10b on remote will cause the remote to emit 

15 volume control commands in accordance with the television's 12 command protocol, thereby 
changing the volume of the television. As another example, when the stereo tuner/amplifier 
16 is the selected device, depressing volume control buttons 10b on remote 10 will cause the 
remote 10 to emit volume control commands in accordance with the stereo's 16 command 
protocol, thereby changing the volume of the stereo. Since the command protocols for the 

20 stereo 16 are different from the television 12, changing the volume of the stereo 16 using the 
remote 10 will not change the volume of the television 12 and vice versa. 

Other control commands well known in the art and to consumers may be sent 
to either the television 12, VCR 14, stereo tuner/amplifier 16 or CD player 18 when the 
particular device is selected in the remote 10. Once the selected device is identified in the 

25 remote, commands input by a user via the button interface on the remote 10 are likewise 
emitted according to the command protocol for the selected device. The format of the 
command protocols used between the remote 10 and the various devices of Fig. 1 are known 
in the prior art and are not described here. The sequence of IR signals comprising a command 
formatted in accordance with a command protocol for a particular device is emitted via IR 

30 light output port lOf at the front face of the remote 10. IR signals are received at IR receiving 
sensors 12a, 14a, 16a, 18a on the faces of television 12, VCR 14, stereo tuner/amplifier 16 
and CD player 18, respectively Command protocols are described in more detail, for 
example, in U.S. Patent No. 5,959,751 to Darbee et al. 
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Selection of the desired device (either television 12, VCR 14, stereo 
tuner/amplifier 16 or CD player 18) to be controlled by the remote 10 and programming the 
remote to use the command protocols corresponding to the selected device are performed 
automatically in accordance with the invention. The device automatically selected by the 
5 remote corresponds to the device at which the remote 10 is pointed by the user. 

In order to establish which device the remote 10 is pointed at, there is a two- 
way communication capability and protocol between the remote 10 and the available devices 
(comprised of television 12, VCR 14, stereo tuner/amplifier 16 and CD player 18), Thus, 
remote 10 not only comprises an ER output port 10f, but also includes an ER light sensor lOg 

10 on the front face of remote 10 that detects incident IR light for decoding and processing by 
the remote 10. In addition, the devices, namely, television 12, VCR 14, stereo tuner/amplifier 
16 and CD player 1 8 each include an IR light output port 12b, 14b, 16b, 18b, respectively, for 
outputting a sequence of IR pulses generated by the respective device. 

IR sensor lOg on the front of remote 10 is a narrow field of view IR sensor, 

15 which are widely available commercially. The field of view of IR sensor lOg is, for example, 
10°. IR light emitted by the output port (either IR light output port 12b, 14b, 16b or 18b) of 
the device at which the user points the remote will be within the field of view of the IR 
sensor lOg (in this case, 10°) and thus is detected by the remote 10. For example, referring 
back to Fig. 1, the remote 10 is shown pointed at television 12. The field of view (of 10°) of 

20 the IR sensor lOg of the remote is shown as cone 20 that intersects the plane of the television 
12 as circle 22. Although the narrow field of view of sensor lOg is represented as symmetric 
about its central axis (resulting in a circular field of view as shown), other shapes of the field 
of view are possible depending on the sensor. In general, however, the narrow field of view 
of the sensor lOg will capture only ER light emitted by the IR light output port of the device 

25 at which it is pointing. Thus, as shown in Fig. 1, IR sensor lOg will only capture light emitted 
by IR output port 12b of television 12, since port 12b is the only port that lies within circle 
22, that is, within the field of view of ER light sensor lOg on the remote 10. 

It is possible that the field of view of IR sensor lOg will envelop the IR output 
port of two devices. For example, in Fig. 1, if the remote 10 were to be pointed at the lower 

30 portion of television 12, the edge of the field of view represented by the circle 22 may also 
envelop the IR output port 14b of VCR 14. However, the ER signal received from the IR 
output port of the device in the center of the field of view (that is, output port 12b of 
television 12) will generally be stronger than the signal received from the device at the edge 
of the field of view (that is, from output port 14b of VCR 14). Thus, remote 10 may also 
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include a filter or the like that removes the weaker IR signal received by the remote 1 0 and 
processes the stronger signal received from the device at which the remote 10 is pointed. 

Each device emits an IR signal from its respective ER light output port that 
provides the appliance code for the device. Thus, television 12 emits a signal via its IR output 

5 port 12b that provides the particular appliance code for television 12. Likewise, VCR 14 
emits a signal via its IR output port 14b that provides the appliance code for VCR 14, stereo 
tuner/amplifier 16 emits a signal via its IR output port 16b that provides the appliance code 
for stereo tuner/amplifier 16b and CD player 18 emits a signal via its IR output port 18b that 
provides the appliance code for CD player 18. 

10 The signals emitted by each device providing its respective appliance code is 

in a standard format that may be decoded by remote 10 via internal processing when 
received. Referring to Fig. lb, a number of exemplary components of the remote of Fig. la 
that provide the processing of the received IR signal are represented. Internal processing may 
be performed, for example, by microprocessor lOp with appropriate attendant software or 

15 other digital algorithms. Microprocessor lOp receives input from ER light sensor lOg and 
input buttons 10a- lOe and provides control signals (which may be via intermediary devices, 
such as an IR LED, not shown) for IR output port lOf. Microprocessor lOp interfaces with 
database 10s, which may be internal to microprocessor lOg and may also provide temporary 
storage for microprocessor lOp. 

20 Although all devices in the room of Fig. 1 emit an IR signal from IR light 

output ports 12b- 16b, as described above, however, because of the narrow field of view of 
the sensor lOg (along with other electronic processing if necessary), the remote 10 will only 
receive and process the IR signal emitted by the device at which it is pointed. Thus, for 
example, if remote 10 is pointed at television 12 as shown in Fig. 1, IR sensor lOg of remote 

25 10 will only receive and process the IR signal giving the appliance code of the television 12 
emitted via its IR output port 12b. Processor lOp of remote 10 then decodes the appliance 
code of the television 12 received from IR light sensor lOg. Using the appliance code for the 
television 12, the associated command protocols for the television 12 are retrieved from the 
comprehensive database 10s of devices in the remote 10. Any control command input by the 

30 user via the button interface lOa-lOe of the remote (shown in Figs. la and lb) are thus 

formatted by processor lOp according to the command protocols for the television 12 and 
output by the remote at the IR output port lOf. 

Thus, remote 10 automatically outputs commands formatted for the device at 
which the remote 10 is pointed. If remote 10 is moved by the user so that it is pointed at 
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another device, for example, the CD player 18, remote 10 receives (via sensor lOg) and 
processes the appliance code for the CD player 18 emitted by the IR output port 18b of CD 
player 18. As described above, because of the narrow field of view of sensor lOg (along with 
other electronic processing to eliminate any weaker IR signals, if necessary), the sensor lOg 
5 of remote 10 will only receive and process the appliance code it is currently pointed at, 
namely the CD player 18. The internal processing of the remote 10 decodes the appliance 
code of the CD player 18 received and uses the appliance code to retrieve the associated 
command protocols for the CD player 18. Any commands input by the user via the buttons 
on the remote 18 will be formatted to control the CD player 18, at which the remote 10 is 
10 pointed. 

In this manner, the user may rapidly control successive devices without the 
need to first select each device on the remote before inputting a command. For example, the 
user may simply point the remote 10 at the television 12 and depress power button 10a to 
turn the television 12. As described above, by virtue of pointing the remote 10 at the 

1 5 television 12, the remote will emit the proper command ("toggle power") according to the 
command protocol for the television 12. The user may then point the remote 10 at the CD 
player 1 8, depress the power button 1 0a to turn the CD player 1 8 on, and then press the play 
button (located in group lOe) to begin play of a CD. Again as described above, by virtue of 
pointing the remote 10 at CD player 18, the remote will emit the proper commands ("toggle 

20 power", followed by "play") according to the command protocol for the CD player 1 8. 

The devices may emit IR signals representing their appliance codes 
periodically at short intervals, for example, such that the remote 10 detects at least one 
emitted signal containing the appliance code from a device in less time than it takes the 
typical user to point the remote 10 at the device and input a control command for the device 

25 via the buttons on the remote. Alternatively, when the user points the remote 10 at a device 
and inputs a control command via the remote 10, the remote 10 first initiates a set-up 
procedure. For example, the remote 10 may emit via its IR output port lOf an "Identify 
Request" signal in a standard format. Any device that receives the Identify Request signal 
over its IR receiving sensors (such as sensors 12a, 14a, 16a, 18a in Fig. 1) responds by 

30 emitting its appliance code. As described above, because of the narrow field of view of the 
sensor lOg (along with other electronic processing if necessary), remote 10 will only receive 
and process the appliance code emitted by the device at which it is pointed. The remote 10 
selects the command protocol from its database for the identified appliance code and 
transmits the control command input by the user formatted according to the command 
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protocol. Accordingly, the device at which the remote is pointing receives the command 
input by the user in the appropriate command protocol. 

In an alternative variation of the embodiment shown in Fig. 1, remote 10 
includes camera optics that capture and provide images to the remote's processor. Thus, as 

5 shown in Fig. 2, remote 10 appears substantially as in Fig. la, except the IR sensor lOg (and 
associated processing for decoding the IR signals containing the appliance code emitted by 
the devices) is omitted and camera optics lOh are present on the top end of the remote 10. 
The optic axis (OA in Fig. 2) of the camera optics lOh lies along the length of the remote 10, 
so the user also points the optic axis of the camera lOh at the desired device when the remote 

10 is pointed at the device. The device intended to be controlled by the user will therefore be 
located substantially in the center of the image captured by the camera optics lOh. 

As shown in Fig. 2a, remote 10 of Fig. 2 comprises substantially the same 
internal components as in the embodiment shown in Fig. lb. In Fig. 2a, of course, processor 
lOp of remote 10 is connected with camera optics lOh. In addition, the software (or other 

1 5 digital algorithms executed by processor 1 Op) comprises image recognition software that 
supports the embodiment and database 10s stores data as further described below. 

When an image is captured by camera optics lOh and received by the 
processor lOp, image recognition software executed by processor lOp focuses on identifying 
an object in the center of the image, where the device is likely to be located. The 

20 comprehensive database 10s of command protocols associated with various devices in the 
remote 10 of this embodiment includes a detailed digital representation of the face of each 
device that is associated with the command protocols for the device. The detailed digital 
representations of the faces of each device, for example, comprise templates of the face of 
each device that are used by the processor of the remote 10 in an image recognition 

25 processing routine to identify a device that is located at the center of an image that is 

captured by the remote. When a match is detected between a stored template for a device and 
the object in the center of the captured image, the command protocols associated with the 
stored template for the device are used by the remote. 

For example, Fig. 3 depicts the image captured by the camera optics lOh of 

30 remote 10 when pointed as shown in Fig. 1. (The field of view of the camera optics lOh is not 
necessarily a narrow field of view as for the sensor lOg of the prior embodiment; thus, as 
shown the image shown in Fig. 3 may be different from the dashed circle of Fig. 1 .) Of 
course, television 12 is shown to be in the center of the image of Fig. 3, since the remote 10 
is pointed at the television 12 in Fig. 1. Referring to Fig. 4, the comprehensive database of 
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templates of devices stored in database 10s of remote 10 of Fig. 2a is represented. The 
templates represented in Fig. 4 are, of course, stored digitally in the remote. The templates 
are a collection of the templates of the front faces of many devices available from various 
manufacturers. For example, as shown in Fig. 4, templates are stored for various models and 
5 makes of televisions (a number of which are represented in Fig. 4 as televisions 100a, 100b 
and 100c) and VCRs (a number of which are represented in Fig. 4 as VCRs lOOd, lOOe and 
lOOf). As represented by the series of dots ("...") in Fig. 4, templates for many other models 
and makes of televisions, VCRs and other devices (such as CD players, stereo 
tuner/amplifiers, DVD players etc.) are also stored in the database. 

10 As noted above, the templates as represented in Fig. 4 that are stored digitally 

in the remote 10 are a collection of the templates of the front faces and associated command 
protocols of many devices available from various manufacturers. Typically, the remote 10 
will come with the comprehensive collection of templates and associated command protocols 
for many commonly available devices pre-loaded in database 10s. In addition, for a device 

1 5 that is introduced after the sale of the remote (or for a device that is otherwise not included in 
the database 10s), the template and command controls may be available for downloading to 
the user's PC from a diskette or CD that comes with the device, or from a manufacturer's 
website. The template and command protocols may then be transferred from the user's PC to 
the database 10s (or a separate programmable non-volatile memory in the remote) via a USB 

20 connection, for example. 

When processor lOp of remote 10 of Fig. 2a receives a captured image from 
camera optics lOh, it attempts to match a template in its database 10s with the object in the 
center of the captured image. Initially, processor lOp may attempt to identify the shape of the 
device in the center of the image in order to reduce the number of comparisons that need to 

25 be made with the stored templates. Thus, for example, processor lOp may first determine that 
the object in the middle of the image of Fig. 3 is approximately a square shape. Having made 
the initial determination, processor lOp then attempts to match templates in its database 10s 
having a square shape (among them, televisions 100a, 100b, 100c of Fig. 4) with the object. 
When a template is found in the database 10s that best matches the object in the center of the 

30 image, the command protocol associated with the best matching template is used by the 
remote to provide commands. 

For example, after comparing the templates of television devices in the 
database 10s, processor lOp may determine that template 100b of Fig. 4 stored in database 
10s best matches the television 12 at the center of the image of Fig. 3. (The processor may 
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have to digitally rotate and scale the template or the object in the image to make the 
comparisons.) Having determined that the television 12 corresponds to the model and make 
of the television represented by template 100b in the database 10s, processor lOp uses the 
command protocols associated in the database 10s with that template (which represents a 
5 particular make and model television) when a command is input by the user. 

The image recognition software may be adapted from the method of detecting 
pedestrians and traffic signs as described in "Real-Time Object Detection For "Smart" 
Vehicles" by D.M. Gavrila and V. Philomin, Proceedings of IEEE International Conference 
On Computer Vision, Kerkyra, Greece 1999 (available at www.gavrila.net), the contents of 

10 which are hereby incorporated by reference herein. A template hierarchy is capable of 
capturing a variety of object shapes in the image by extracting a binary feature image and 
creating a Distance Transform ("DT") image thereof. The binary templates (as represented, 
for example, in Fig. 4) are transformed into DT templates before comparison with the DT 
image. One of a number of matching measures are used to determine a match, for example, a 

15 match may be found when the average distance to the nearest feature is a minimum or below 
a user supplied threshold. 

In an improvement of such matching processing described in the "Real-Time 
Object Detection For "Smart" Vehicles" document, matching is achieved using a variant of 
Distance Transform based-matching, that uses a simultaneous coarse-to-fine approach over 

20 the shape hierarchy and over the transformation parameters. Thus, similar templates are 

grouped together, for example, square templates (100a, 100b, 100c, etc.) for the televisions of 
Fig. 4, rectangular templates (lOOd, lOOf, lOOg, etc.) for the VCRs, as well as other 
appliances. Each group of shapes is represented by a prototype template and matching with 
the DT image is first performed using the prototypes, rather than the actual templates 

25 comprising the group. After determination of the best match with a prototype template, the 
individual templates comprising the group stored in database 10s are compared with the DT 
image. 

For example, a prototype DT template of a square (generically representing 
televisions), rectangle (generically representing VCRs), etc. may first be compared by the 
30 processor lOp of the remote 10 with the DT image of the television 12 of Fig 3. The 
prototype DT template of a square will have the best match with the DT image of the 
television 12 of Fig 3. After such a determination, the processor lOp uses the individual 
(square) templates of the televisions 100a, 100b, 100c, etc. of Fig. 4 stored in database 10s 
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that comprise the group represented by the square prototype for further comparison and 
determines the best match. 

The actual templates may also be further grouped into sub-classifications of 
similarities and each sub-classification may be represented by a prototype template that 
5 represents the similarity of actual templates in the sub-classification. By matching prototype 
templates for successive sub-classifications of similarities with the DT image, comparing the 
number of templates that represent actual devices with the DT image is reduced to those 
templates comprising the group of the prototype template for the final sub-level. 

For example, televisions may be sub-classified as having a square screen or a 

10 oval screen. After it is determined that a square prototype template matches the DT image of 
the television in Fig. 3, a second level (sub-classification) of square prototype templates may 
be compared with the image. Two prototype templates may be in the second level, one 
having a square boundary and a square interior border (representing a square screen) and 
another having a square boundary and an oval interior border (representing an oval screen). 

15 Televisions 100b, 100c of Fig. 4 (having square screens) would be among those comprising 
the group represented by the prototype template with the oval interior boundary and 
television 100c of Fig. 4 (having an oval screen) would be among those actual templates 
comprising the group represented by the prototype with the oval interior boundary. After 
comparing the DT image of the television 12 of Fig. 3 with the prototype templates in the 

20 second level, the processor lOp will conclude (based on the relative measures of matching) 
that the prototype template with the square boundary is the best match with the DT image of 
Fig. 3. After such a determination, the processor lOp retrieves from database 10s the 
templates of the televisions 100b, 100c, etc. comprising actual devices in the group 
represented by the matching second level prototype for further comparison and determines 

25 the best match. 

Once the processor lOp of the remote 10 determines which template stored in 
the comprehensive database 10s matches the object in the image, the command protocols 
associated with the template (which represents a particular make and model device) are used 
by the remote 10 when commands are input by the user. For example, as previously noted, 

30 after the stored templates of Fig. 4 are compared with the image of the television 12 in Fig. 3, 
the processor lOp in the remote 10 determines that template 100b matches the object 
(television) 12 in the center of Fig. 3. Command protocols for the make and model of 
television 12 represented by template 100b are associated with template 100b in the database 
10s and these command protocols are used by the processor lOp when the user inputs a 
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command to the remote 10. Thus, by pointing the remote 10 at the desired device, the remote 
automatically identifies the command protocol for the device and emits commands using the 
identified command protocol. 

In addition, once a number of templates in the database 10s have been 
5 identified by the remote 10 as corresponding to devices that are selected by the user, the 
processor lOp may first compare those previously identified templates with the image 
captured and determine whether the matching measure for one of the previously identified 
templates is approximately the same as when the device was last identified. If one of the 
smaller number of previously identified templates matches the object in the image to 

10 substantially the same degree, the processor lOp determines it is a match and uses the 
command protocols associated with the matching template. Also, the stored templates 
corresponding to devices in a room may be identified and updated by the user initiating a 
learning procedure by depressing a button on the remote 10. When the user points the remote 
at a device in the room and initiates the learning procedure, the remote matches the image of 

15 the device with a template and associated command protocols for the device, as described 

above. The learning procedure is conducted for each device the user wants to control, and can 
be updated when new devices are added or removed. After such a learning procedure is 
performed, when the user points the remote at a device, the processor lOp only considers 
those templates for the devices previously identified when determining a match. This reduces 

20 the response time of the remote considerably. Also, because the remote only chooses a match 
for a device among templates for devices owned by the user, the matching is faster and more 
reliable. 

Alternatively, devices may come with an ascribed feature, that may be readily 
identified in an image processing routine. A bar code (or the like), for example, may be 

25 affixed to the front of each controllable device. In that case, the comprehensive database 10s 
in the remote 10 will include a bar code for each device and the device's corresponding 
command protocols. The image captured by the camera of the remote is searched for a bar 
code and, once found, the bar code is analyzed and decoded. Once decoded, the command 
protocols for the identified device are retrieved from the database and used. 

30 As described for the first embodiment of the invention, the remote may 

capture images via camera optics lOh periodically and apply the image recognition 
processing to match a device located in the center of the image. The time interval between 
successive capture and processing of an image would be less than the average time it would 
take a user to enter a command after pointing the remote. Alternatively, the input of a control 
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command by the user using the buttons of the remote may initiate the image capture and 
processing for the device at which the remote is pointed, as described above. Once the 
command protocol for the device in the center of the captured image is identified, the control 
command input by the user is emitted according to the command protocol. 

The second variation of the invention described immediately above, which 
relies on image processing in the remote to identify the device at which the remote is 
pointing, and the first variation of the invention previously described, which relies on the 
device emitting an IR signal giving its appliance code, may also be implemented together in 
the same universal remote and system. In that case, remote 10 will be a hybrid of the remotes 
10 of Figs, la and 2, namely, it will have both an IR light sensor lOg and camera optics lOh. 
One determination of the make and model of the appliance (made, for example, using an 
appliance code received via an IR signal) may be used to confirm another determination 
(made, for example, using template matching in an image processing routine). Where the 
device determinations made using the IR signal and image processing do not agree, an error 
subroutine may be invoked by processor in the remote 10. For example, the template 
corresponding to the appliance code of the IR signal may be retrieved from the database and 
compared with the object in the image. If its measure of matching is high, the command 
protocols for the appliance code of the IR signal are used. If not, then the command protocols 
associated with the template determined via the image processing routine are used. 

In addition, it was previously noted that the user's input of an actual command 
using the buttons on the face of the remote could initiate the determination of the device at 
which the remote is pointed. Alternatively, the remote may include a separate setup button 
that the user may depress when pointing the remote at a device, thereby initiating the 
determination of the command protocols for the device at which the remote is pointed (via an 
IR signal giving the appliance code of the device or image recognition processing, for 
example). In addition, having determined the command protocols for the device, the remote 
10 may use those command protocols until the user again depresses the setup button on the 
remote. In that case, the user can continue to control the same device even though the remote 
10 may not be pointed directly at the device. Should the user wish to change devices, the 
remote 10 is pointed at the next device to be controlled and the setup button is again 
depressed. 

In both variations of the embodiment above, discriminating between two (or 
more) devices detected by the remote 10 were described. In the first variation it was noted 
that filtering or like electronics could be used by the remote 10 where multiple IR signals 
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having appliances codes were received by narrow field sensor lOg. It was presumed that the 
IR signal from the device at which the remote is pointed would emit the stronger signal. In 
the second variation it was noted that the image recognition processing would create a DT 
image for the object most closely located in the center of the captured image. If two or more 

5 objects were captured (in whole or in part) in the image, the device at which the user was 
pointing the remote would most likely appear in the center of the image. 

However, it may often be the case that two or more devices are in close 
proximity and/or the remote 10 is not precisely aimed at the desired device by the user. For 
example, if the remote 10 in Fig. 1 is aimed at the bottom of television 12, IR output ports 

10 12b, 14b of television 12 and VCR 14 would both be located well inside the narrow field of 
view 22, so IR signals emitted by television 12 and VCR 14 detected by the remote 10 may 
have approximately equal strength. (Also, the strength of the IR signals emitted by different 
devices may be different or the device at the center of the field of view may be further away 
than the device at the periphery; thus, the IR signal from the device at which the remote 10 is 

15 pointing may be weaker than the nearby device.) If image recognition is used by the remote 
10 to identify the device, it may also be difficult for the processing to identify whether the 
television 12 or the VCR 14 is located in the "center" of the captured image. 

Where the remote may be pointing at one of two or more devices, the remote 
10 may identify all of the devices to the user and require the user further select which device 

20 is intended to be controlled. For example, if the remote 10 is pointed such that it receives IR 
signals of the appliance codes for both the television 12 and the VCR 14 in Fig. 1, the 
processing within the remote 10 may look-up and retrieve a description of the make and 
model of both devices in the database using the appliance codes received as described for the 
first variation above. The remote 10 may then display a description of the devices on an LCD 

25 or other display in the remote 10. For example, the remote may display "Toshiba model M6 1 
television" for television 10 and "Panasonic model PM3 VCR" for VCR 12 on the remote's 
display. The user selects which device is intended to be controlled via a button or other input 
to the remote. Once selected, the processor retrieves the command protocols from the 
database for the selected device (in the example, either the TV 12 or the VCR 14) and uses 

30 them for any commands input by the user. 

Similarly, for example, if the remote 10 is pointed such that it cannot be 
determined whether the television 12 or the VCR 14 are located at the center of an image, the 
processing within the remote 10 may perform image recognition (using template matching, 
for example) for both objects in a central region of the image. The make and model of 
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devices stored in the database associated with templates that best match the objects in the 
image are determined as described for the second variation above. The remote provides the 
description of the makes and models of the identified devices to the user and the user selects 
the intended device. Once selected, the remote uses the command protocols in the memory 
that are associated with the selected device. 

Referring to Fig. 5, a general flowchart of a method according to the invention 
is given. In step 100, a particular device is selected from among a number of devices based 
upon a selection direction. For example, in the case of the remote of the prior embodiments, 
the selection direction is the direction in which the user points the remote. In step 105, 
identification data is acquired from the particular device. This may be, for example, an IR 
signal containing the appliance code transmitted from the device, or may be a captured image 
of the device, as in the previously described embodiments. In step 110, command protocols 
associated with the particular device are determined using the identification data. Where, for 
example, the identification data is an IR signal containing the appliance code, the appliance 
code may be used to consult a compilation of appliance codes for devices and their associated 
command protocols. The command protocols associated in the compilation with the 
appliance code of the particular device is selected. Where the identification data comprises an 
image of the particular device, the image may be used to find a matching image 
representation in a compilation of image representations for devices and their associated 
command protocols. The command protocols associated in the compilation with the image 
representation that matches the image of the device is selected. In step 120, control 
commands for the particular device are formatted according to the determined command 
protocols. 

Although illustrative embodiments of the present invention have been 
described herein with reference to the accompanying drawings, it is to be understood that the 
invention is not limited to those precise embodiments. For example, the command protocols 
associated with an identified device have been described as being retrieved from a 
comprehensive database in the remote once the device at which the remote is pointing is 
identified. Alternatively, for example, the remote may receive a wireless download of the 
command protocols from the device at which it is pointing, for example, via the narrow field 
IR light sensor lOg of the previously described embodiment. Thus, it is intended that the 
scope of the invention is as defined by the scope of the appended claims. 
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1 . A remote (10) that is capable of controlling a number of devices (12, 14, 16, 
18), the remote (10) acquiring identification data from a particular device (12) at which it is 
pointed, processing the data to determine command protocols associated with the particular 
device, and formatting control commands for the particular device (12) input to the remote 

5 (10) by a user according to the command protocols associated with the particular device (12). 

2. The remote as in Claim 1, wherein the remote (10) acquires identification data 
from the particular device (12) at which it is pointed by receiving a signal emitted from the 
device (12) at which it is pointed. 

10 

3. The remote as in Claim 2, wherein the remote (10) comprises a sensor (lOg) 
that detects the signal emitted by the particular device (12) in a narrow field of view about 
the direction in which the remote (10) is pointed. 

15 4. The remote as in Claim 2 or 3, wherein the signal is an IR signal that includes 

the appliance code of the particular device (12). 

5, The remote as in Claim 1, wherein the remote (10) comprises a database (10s) 
that comprises identification data of the number of devices (12, 14, 16, 18) and their 

20 associated command protocols, the remote (10) using the identification data of the particular 
device (12) to determine the command protocols associated with the particular device (12). 

6, The remote as in claim 1, wherein the remote (10) acquires identification data 
from the particular device (12) at which it is pointed by capturing at least one image in the 

25 direction in which the remote (10) is pointed, the at least one captured image including the 
particular device (12), and by identifying the particular device (12) in the image using image 
recognition processing. 
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7. The remote as in Claim 6, wherein the remote comprises a database (10s) that 
includes representations of images (lOOa-lOOf) of the number of devices (12, 14, 16, 18) and 
their associated command protocols, the remote (10) determining a representation of an 
image (100b) in the database (10s) that matches the particular device (12) in the image, the 

5 remote (10) using command protocols associated with the representation of the image (100b) 
in the database (10s) to format input control commands. 

8. The remote (10) as in Claim 1, wherein the remote (10) acquires identification 
data from the particular device (12) at which it is pointed after the user inputs an acquisition 

10 command in the remote (12), wherein, after processing the identification data acquired to 
determine command protocols associated with the particular device (12), the remote (10) 
formats control commands for the particular device (12) input to the remote (10) by the user 
according to the command protocols associated with the particular device (12) until the user 
inputs another acquisition command. 

15 

9. The remote as in Claim 1, wherein the remote (10) acquires identification data 
from two or more particular devices (12, 14) at which it is pointed, the remote (10) 
processing the identification data and displaying the identity of the two or more particular 
devices (12, 14) to the user, the user providing a selection input to the remote (10) selecting 

20 one of the two or more particular devices (12, 14) identified, the remote (10) determining 
command protocols associated with the selected device (12 or 14) and formatting control 
commands for the selected particular device (12 or 14) input to the remote (10) by the user 
according to the command protocols associated with the selected particular device (12 or 14). 

25 10. A method for controlling one particular device (12) selected from among a 

number of devices (12, 14, 16, 18) based upon a selection direction, the method comprising 
the steps of: acquiring identification data from the particular device (105), determining 
command protocols associated with the particular device using the identification data (1 1), 
and formatting control commands for the particular device according to the determined 

30 command protocols for the particular device (1 20). 

1 1 . The method as in Claim 10, wherein the step of acquiring identification data 

from the particular device (105) comprises receiving an IR signal containing the appliance 
code transmitted from the particular device. 
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